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ABSTRACT 

The purpose of this paper is to determine potential identifiers of students’ academic success in foundation 
mathematics course from the data logs of the intelligent tutor Assessment for Learning using Knowledge Spaces 
(ALEKS). A cross-sectional study design was used. A sample of 152 records, which accounts to approximately 
60% of the population, was extracted from the data-logs of the intelligent tutor, ALEKS. Two-step clustering, 
correlation and regression analysis, Chi-square analysis and ANOVA tests were applied to address the research 
questions. The data-logs of ALEKS include information about number of topics practiced and number of topics 
mastered by each student. A derived attribute, which is the ratio of number of topics mastered to number of 
topics practiced is found to be a predictor of final marks in the foundation mathematics course. This variable is 
represented by the name mtop. Cluster classification based on this derived attribute resulted into three groups of 
students for which the mean values of the variable mtop are 0.80, 0.66 and 0.53 respectively. A moderately 
strong, positive and significant correlation was found between mtop and the final exam marks. 

Keywords-cluster analysis, intelligent tutor, learning analytics, ALEKS 

Introduction 

It has been reported that many students spend a greater period of time in early years of the higher education 
especially in remedial or foundation programs; they spend more than expected without achieving the program 
requirements (Nasser, 2012). As Hansen et al (2006) explain, that secondary schools prepare students for 
university requirements but do less in preparing them to achieve at the level that universities require. 
Significantly, English language appears to be a difficult subject as most universities use that medium of 
instruction to teach science and non science subjects. Many students in the Gulf countries have not achieved a 
level of competency to enable them to successfully operate in English as a language of instruction and learning. 

Many programs, techniques, and methods are in place to support foundation year students in their learning in 
higher education among them is continuous assessments and review which are probably one of the key 
ingredients to improve student learning strategies to be used for enriching their learning experience as it has the 
cognitive as well as the motivational purpose (Ritter, Anderson, Koedinger & Corbett, 2007). Timely feedback 
in assessments fills the gaps between the actual learning and expected learning outcomes (Chappuis, 2014). 
Formative assessment with feedback is particularly significant when students are uncertain about what is 
expected of them and when they need instructional guidance about how to move ahead (Nguyen, Hsieh & Allen, 
2006; Wood & Wood, 1996). One claim is that the process by which students are monitored during instruction 
can help teachers provide timely feedback on students’ actual learning. 

While formative assessments or continuous assessments followed by feedback can be used periodically to assess 
students’ learning, it may not be feasible or practical to incorporate in large class sizes, (Chappuis, 2014). The 
first years in higher education, a great deal of stress is placed on the instructor to carry out continuous 
assessments, or formative assessments and provide timely feedback. More lately the use computer-based 
assessment systems known as intelligent tutoring are widely used in secondary schools and higher education. 
These systems are web-based and designed to run on multiple devices such as laptops, iPads and mobile gadgets. 
They can be used to conduct frequent formative assessments with appropriate and timely feedback to minimize 
the gap between actual learning and expected learning (Narciss & Huth, 2004). It also engages students in 
authentic learning opportunities and can increase student participation and motivation in the learning process 
(Miller, 2009). 
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A key feature of these software systems is their ability to record and store every learning activity occurring when 
a student interacts with the system. The data gathered for every user can be analyzed providing the “learning 
profiles” for each student or at the aggregate level. A learning profile is useful to understand students’ study 
habits and their progress (Kotsiantis, Tselios, Filippidi & Komis, 2013). Learning profiles can be detected by 
applying methods of Learning Analytics in which system-generated large data logs are analyzed in order to 
understand students’ learning activities (Siemens & Long, 2011). The data generated support instructors to 
assess where the students are and where to go forward. 

Literature review 
Computer based assessments 

In class, the instructor can engage students in student-centered activities in the classroom through intelligent 
tutors. Students can access such systems online at any time or anywhere, for which they are expected to develop 
self-regulatory approaches to succeed and use the technologies available to them in and outside the classroom 
(Aleven, Roll, McLaren, Koedinger, 2010; Nicol, 2006; Nguyen, Hsieh & Allen, 2006). Web-based intelligent 
tutoring systems allow students to practice, to have control over their learning and manage their time and 
interaction with peers and instructors (McArthur & Stasz, 1990). Also the use of online computer based 
assessments, have several advantages over paper based assessments (Aleven, Roll, McLaren & Koedinger, 2010; 
Balacheff & Kaput, 1996; Hagerty & Smith, 2005), they provide access to any number of students anytime and 
anywhere through any type of computer, such as a laptop, tablet or a smart phone. They provide a wider range of 
assessment techniques than the paper based assessments, for example, inclusion of graphics and multimedia. 
Students can provide their responses in various formats, such as drawing graphs on a digital screen, locating 
number positions on a number line by clicking on the webpage. The most significant aspect of computer based 
assessments is that individualized feedback is given instantly. More importantly, the software applications can 
generate questions randomly from a large bank of questions and different versions of assessments produced to 
tailor different levels of learning outcomes and practice questions required for mastering a topic (Shute & 
Underwood, 2006). Moreover, such web-based software can foster student-centered learning by engaging 
students in meaningful learning activities and can increase students’ engagement in learning (Chen et al, 2008; 
Chen, Yunus, Ali & Bakar, 2008; Nguyen, Flsieh & Allen, 2006; Schneider, Egan & Julian, 2013). 

Intelligent tutors 

Generally, a computer tutor or intelligent tutor establishes task-related goals and guides the learner toward the 
goal. An expert tutor is capable of designing learning tasks to ensure that the student persists on the task and 
gains some new knowledge, the student when interacting with the computer tutor may heavily rely on the system 
to work out a problem as an act of educational transference (Thelwall, 2000). While early computer-based 
learning was based on behaviorist learning theories that each application is taught as a separate learning 
objectives, into modules with separate objectives that are linked in such a way that the outcomes in one module 
can be used as an input into another. However more recent advances in cognitive sciences recommend adapting 
the constructivist learning theory and emphasize that “true” understanding as connected and generalizable 
knowledge wholes (Ritter, Anderson, Koedinger & Corbett, 2007). According to constructivist learning 
paradigm, a student could cultivate independent and self-directed learning and higher ordered thinking. 
Researchers as Chen, Yunus, Ali & Bakar (2008) and McArthur & Stasz (1990) have shown that computer- 
based or web-based assessments had positive effects on students' mathematical learning processes especially 
where problems required analytical and critical approaches for solving them. In mathematics immediate 
correction and feedback can generally have substantial real-time benefits to students as it gives them an 
opportunity to analyze the problem and readjust, reorganize, restate, and recalculate the problem work and move 
to higher levels of the taxonomy of educational objectives. 

The current and emerging technologies, such as intelligent tutors, which are supported by artificial intelligence 
techniques have an advantage compared to other information technologies (Chen, Yunus, Ali & Bakar 2008; 
Chen et al, 2008; McArthur & Stasz, 1990; McGatha, & Bush, 2013). The intelligent tutors have the ability to 
integrate more than one medium, provide authentic and concurrent learning activities and provide academic- 
content based support to a large student body. As reported in (Stiggins, 2001; VanLehn, 2011) human tutoring 
has an effect size of d = 2.0 relative to classroom teaching without tutoring. This effect is known as the ‘two 
sigma gain’. Developers of intelligent tutors work towards achieving the same effect as human tutors by 
incorporating multidimensional tutoring with appropriate feedback and scaffolding techniques based on the 
knowledge of the subject and the knowledge of student’s state of learning (Kao & Lehman, 1997; Stiggins, 
2001). Intelligent tutors’ development is based on combining theories of cognitive science and techniques of 
artificial intelligence (Anderson, Boyle, Corbett & Lewis, 1990; Ritter, Anderson, Koedinger & Corbett, 2007; 
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McGatha & Bush, 2013; Miller, 2009). The intelligent tutors can provide interactive and personalized learning 
environment for students allowing them to study and learn individually (Hagerty & Smith, 2005). 

Some intelligent tutoring system, such as Cognitive tutor, allows students to write solutions procedurally as if 
they were solving it on paper. The system gives feedback on each step as well as for the overall solution (Ritter, 
Anderson, Koedinger & Corbett, 2007) whereas, intelligent tutors like ALKES (Assessment for Learning using 
Knowledge Spaces), provides feedback only on the final answer. Cognitive tutors are appropriate for novice 
learners where every step is supported through feedback, use of systems like ALEKS is appropriate in higher 
education where students are expected to develop the ability to follow through problem-solving procedures with 
minimal support. 

One of the prominent theoretical frameworks underlie the development of intelligent tutoring systems, is the 
framework of knowledge space theory. The knowledge space theory is applied to make the learner agile to 
learning. Tutoring systems, such as ALEKS, is built on the foundations of knowledge space theory that can 
gauge the level of student’s understanding and can detect the correctness of student’s next response on the basis 
of current response. ALEKS provides learning goals, scaffolding support for learning and allows for formative 
and continuous assessments and feedback. 

At the core of the analytic engine is the concept of two fringes. One fringe which consists of all topics that What 
a Student Can do and the second fringe consists of all topics that the student is Ready to do or Learn. Refer to 
Table 1 for illustration. 


Table 1: Two states of student’s learning (excerpt only) 


What H00298326 Can Do as of 09/15/2014 

What H00298326 Is Ready to Learn as 
of 09/15/2014 

Place Value, Expanded Form, and Numeral 
Translation 

Numeral translation: Problem type 1 

Exponents and Order of Operations 

Writing expressions using exponents 


ALEKS is user friendly and interactive. A student can choose any topic available from the list of ‘Ready to 
Learn Topics.’ A question is presented on that topic by the system, a student can request an explanation and if a 
student can respond to the problem correctly, positive reinforcement is prompted on the system. If the student 
can answer three more similar questions correctly, then the system allows the user to terminate the task by 
prompting the option of ‘Done.’ If a student is confident about the mastery of this topic then, they can click on 
the button ‘Done,’ and the topic is added to the list of ‘what a student can do.’ If a student cannot answer three to 
four consecutive questions correctly, then the system does not present questions from the same topic but suggests 
that the student can try another topic. ALEKS has the ability to create individualized sequence of topics based on 
the student’s background knowledge and level of cognitive development but the instructions provided by 
ALEKS are static and same for all students irrespective of their individual learning styles. It does not provide 
instructions in different multi-media format, such as audio or video, but allows instructor to upload presentations 
and video files customized for students. 

ALEKS sets two types of in-built and individualized assessments known by the progress test and comprehensive 
test. These assessments include questions from the two sets, the first, a set of topics mastered by the student and 
the set of topics which the student is ready to learn. Progress tests are administered by the system based on the 
topics mastered and time spent by the student, whereas comprehensive tests must be assigned by the instructor. 
The purpose of the progress test is to ensure that students can retain and recall his or her learning. Thus by 
diagnosing student’s current state of knowledge, the software can provide scaffolding exercises and/or problems 
that help the student progress gradually. Each student can learn at her own pace and monitor her own progress. 
Inclusion of ALEKS in the foundation mathematics curriculum is aligned with the strategic decision of ministry 
of higher education in the United Arab Emirates (UAE) to basically integrate computer-based technologies in the 
educational processes. 

Learning Analytics 

Learning analytics focus on deriving information which can reveal how students use the intelligent tutoring 
systems and identify potential “identifiers” of academic achievement. (Desmarais & Baker, 2012; Holden, 
Sottilare, Goldberg & Brawner, 2012; Kotsiantis, Tselios, Filippidi & Komis, 2013; Libbrecht, Rebholz, 
Herding, Muller & Tscheulin, 2012). Application of methods of learning analytics can be a powerful means to 
inform and support learners, teachers and their institutions to better understand and predict individualized 
learning needs and performance (Greller & Drachsler, 2012; Siemens & Long, 2011, Tempelaar, 2014). There 
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are specific student attributes when analyzing learning patterns, such attributes include time - spent on a topic, 
engagement with it and other system dispositional elements as skills and computer agility (Siemens & Long, 
2011). Such system specific attributes are taken into account for analyzing students’ learning patterns or their 
engagement in learning, but in some cases new attributes are derived to gain deeper understanding of 
determinants of students’ learning (Antonenko, Toy & Niederhauser, 2012). Learning analytics attributes can be 
derived by combining the information about the number of topics practiced, time spent on learning and number 
of topics mastered. This analysis could provide information about whether a student can learn from the 
instructional queues and feedback provided by the software and utility for mastering the course content. 

The system-specific attributes generated by ALEKS may not provide accurate information about student’s 
learning efforts as the system cannot indicate the idle time, when students login to the system and do not attempt 
to respond. In addition, the time taken to master a topic is not signified as students are encouraged to learn at 
their own pace. It is worth investigating how to detect from such large data logs, information about students who 
are able to master a topic by studying independently. 

We calculated the ratio of the two variables number of topics mastered and number of topics practiced, 
represented by the variable mtop (which is an abbreviation of mastered-to-practiced) which can be used as a 
construct of the extent of which the student has the ability to learn independently. The aim of this research is to 
examine whether mtop is a predictor of student’s assessment in a course. 

Course Structure 

In the context of the UAE not too long ago, the ministry of higher education of the UAE took a decision to 
supply tablets to foundation year students in all federal higher education institutions. This decision was taken to 
address the strategy to develop technologically advanced environments to support learning in higher education 
(Gitsaky, Robby, Hamdan & Ben-Chabane, 2013). The supply of tablets to the foundation year students as the 
first year experience was perceived as an impetus for students to “ride” the information age and stay abreast of 
the technological advancement in higher education in preparation for the workplace (Nguyen, Elsieh & Allen, 
2006; Yorke & Longden, 2004). 

Two foundation courses covering basic arithmetic, algebra, geometry and statistics are delivered using the 
ALEKS software used tablets. Students use their tablets (iPads) to access this program. The software provides 
explanation and practice problems on each topic. Students are expected to master all topics as per their learning 
pace. Upon registering into the course on ALEKS, the software gives each student an initial assessment and 
detects their prior knowledge about the subject. This score is denoted by the variable Initial Assessment (IA). As 
the student interacts with the software and progresses towards the completion of all topics, the software 
maintains a record of progress and the status of mastery of the course is displayed in the form of a Pie chart as 
shown in Figure 1. 


12/10/2014 Learning 


12/10/2014 Learning 


Last login: 
Enroll date: 
Total Hours: 


12/10/2014 

09/10/2014 Hours/week: 

50 Hours 13 minutes 

Course Mastery 
(349 of 400 topics) 


► Exponents and 
Polynomials (23 of 50) 



Graphs of Linear 
Equations (22 of 22) 

Inequalities (4 of 5) 


► Algebraic Expressions 
and Equations (58 of 65) 


Real Numbers (27 of 27) 


Geometry (71 of 71) 


Numbers (44 of 44) 

Fractions (21 of 21) 
Decimals (14 of 14) 


Ratios, Proportions, 
and Percents (41 of 41) 


► Data Analysis and Measurement (13 of 13) 
Statistics (11 of 27) 


Figure 1: Pie-chart showing learning status of a student on ALEKS 
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In terms of course grade distribution through ALEKS, 40% weighting is assigned to completion of all topics, 
which works as the formative assessment. Students are expected to master these topics outside the regular class 
time. 60% weighting is given to in-class quizzes and the final exam which form the summative assessment 
component is denoted by FE in the rest of the paper. The assessments are created by teachers but graded by the 
software. Students can review their own answers after the examinations are graded, but the software does not 
provide a detailed feedback on their answers. The software only indicates whether the answer is correct or 
incorrect. In case of incorrect answers, the system does not provide an explanation. It only provides the expected 
correct answer. 

There is also a summative assessment, which is a comprehensive test generated by ALEKS, and is based on what 
the student has mastered. This assessment component is denoted by CT in the rest of this paper. These tests are 
conducted in classroom under controlled conditions. After each test, the software indicates which topics are 
retained by the student and which are not. In these tests unlike the fonnative assessments, the software does not 
provide feedback on student’s performance in the comprehensive test, neither teacher can see student’s solutions 
nor the student can see their own answers and know the mistake; hence a student has to re-leam the topic that is 
dropped after each comprehensive test. 

Regular course runs over 16 weeks but students who complete at least 85% of the topics after each 
comprehensive test are given an opportunity of exiting the course earlier than those who do not. The software 
allows teachers to set individual or group classwork, homework, quizzes and worksheets. 

Methods 

Data was gathered from a cumulative report generated from a 20-weeks data which included the following 
information: time spent in ALEKS in each week, number of topics practiced each week and number of topics 
mastered each week. The excerpt of the data fde used for analysis is given below and each variable is described 
subsequently. 


Table 2: Excerpt of the data file 


EPL 

Number 

EE 

WT-1 

WM-1 

WP-1 

mtop- 

WT-1 

WM-2 

WP-2 

mtop- 

(English 

of 

IA CT FE (early (Time 

(Topics 

(Topics 

1 

(Time 

(Topics 

(Topics 

1 

Language 

progress 

exit) 

spent 

mastered 

practiced 

(Ratio 

spent 

mastered 

practiced (Ratio 

proficiency) 

tests 


in 

in 

in 

of 

in 

in week - 

in 

of 


Ptests 


week - 

week -1) 

week-1) 

WM- 

week - 

2) 

week-2) 

WM-1 




1 



1 

1 in 



/WP- 




in 



/WP- 

min) 



1) 




min) 



1) 






3 

6 

36 

83 

71 

Yes 

237 

22 

25 

0.88 

432 

24 

37 

0.65 

3 

5 

27 

61 

61 

No 

38 

4 

4 

1.00 

189 

19 

21 

0.90 

3 

4 

28 

50 

32 

No 

48 

0 

0 

0.00 

251 

25 

34 

0.74 

3 

7 

17 

65 

60 

No 

288 

26 

27 

0.96 

1334 

126 

152 

0.83 

3 

3 

23 

62 

60 

No 

72 

5 

5 

1.00 

365 

30 

31 

0.97 


IA=Initial assessment , CT= Comprehensive test, FE=Final Exam; WM-1: Topics mastered in week-1; WP1- 
Topics practiced in week-1 ,mtop- 1: Ratio of WM-l/WP-1 WM-2: Topics mastered in week-2; WP2-Topics 


practiced in week-2, mtop- 2: Ratio of WM-l/WP-1. 

The data file also included the following variables: student’s score in the initial assessment (IA), total number of 
topics mastered by the student after the comprehensive test (CT), student’s marks in the final exam (FE), and 
number of progress tests taken by the student (Ptest) and whether the student passed the course or not in less than 
12 weeks. If a student passes the course in less than 12 weeks, then the values assigned attribute to variable EE 
is “Yes.” For other students who do not pass in less than 12 weeks, the assigned attribute to this variable is 
“No.” The system administers progress tests based on the number of topics completed by a student. The number 
of progress tests attempted is different for each student as the pace of their learning is different. In the data file, 
the variable Ptest denotes the number of progress tests taken by a student. The ratio of the two variables of topics 
mastered to topics practiced is represented by the variable mtop for each week and is used as a measure of ability 
to learn independently. The mean value of this variable mtop over 20 weeks was calculated. Refer to the Table 2 
given above. 
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This research aims to assess the relation between student’s ability to work independently through ALEKS and 
student’s final marks in the course. The research aims are: 

(1) To explore learning profiles of students based on similar learning patterns. 

(2) To investigate the following research questions: 

Does the ability to work individually effect students’ marks in the coursework and in the final exam? 

Does the proficiency in English affect the ability to study individually? 

Data Analysis 

The data file consisted of 152 records from five sections of Basic Mathematics and Pre-Algebra taken at a 4-year 
technical college in the UAE. The students were in the foundation year, and candidates enter regular degree 
programs upon completion of English and Mathematics courses. 

In the first stage of the analysis the Shapiro-Wilk test of normality was applied to test the normality of the 
variable mtop. The result shows that value of the statistic is 0.99 and p-value is 0.142. Since the p-value is higher 
than 0.05, it implies that the variable mtop is normally distributed and hence parametric tests are applicable. 

Cluster analysis 

In order to determine which groups of students have similar learning profiles, a cluster analysis can be applied 
(Antonenko, Toy & Niederhauser, 2012; Cohen, Manion, and Morrison, 2011). Two-step clustering method is 
applied where variables are continuous and the number of clusters is not known apriori (Field, 2009). Students 
are classified into clusters based on the mean value of the ratio of topics mastered to topics practiced (mtop). The 
clustering created three different profiles based on the value of the variable mtop. The software detected three 
clusters by applying the Log-likelihood method. Based on these cluster profiles, it is observed that the students in 
the cluster number one had the highest value for the variable mtop, which means on an average they mastered 
80% of the topics out of the topics that they practiced, whereas students in the cluster two and cluster three 
mastered only 66% and 53% of topics, respectively. One-way ANOVA test was applied to test if these clusters 
were independent of each other. The results of the ANOVA test showed that the mean value of mtop was 
statistically different for each cluster (F=10.26, p-value=0.000), which confirms that the three clusters are 
independent of each other. 

Table 3 presents the cluster distribution and the mean and standard deviation of the clusters. 


Table 3: Cluster profiles 


Cluster number 

Mean (mtop) 

S.D. (mtop) 

Number of students 

1 (high) 

0.80 

0.05 

32 

2 (Medium) 

0.66 

0.05 

61 

3 (Low) 

0.53 

0.03 

59 


Effect of mtop on early completion of the course. As described in the section above, students were given an 
opportunity to pass the course in less than 12 weeks if they mastered 85% of topics by studying independently. 
A total of 34 students out of 152 passed the course within 12 weeks. Out of those 34 students, the 44% belonged 
to the cluster two which means a high percent of students who passed the course early, were able to master 67% 
of the topics they practiced. Whereas 35% students belonged to the cluster one, which means they were able to 
master 80% of the topics they practiced. A total of seven students in this cluster three passed the course in less 
than 12 weeks, which means they were able to master only 53% of the topics they practiced. 


Table 4: Cross-table showing number of students who passed the course early in each cluster 

Two-Step Cluster Number 





1 (high) 

2 (medium) 

3 (low) 

Total 

Early 

No 

Count 

20 

46 

52 

118 

exit 


% within early exit 

16.9% 

39.0% 

44.1% 

100.0% 


Yes 

Count 

12 

15 

7 

34 
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% within early exit 


35.3% 


44.1% 20.6% 


100 . 0 % 


Total 


Count 


32 


61 


59 


152 


% within early exit 


21 . 1 % 


40.1% 38.8% 


100 . 0 % 


Whereas though students in the cluster one, had a high score for mtop, 20 students from this cluster did not pass 
the course early. Refer to Table 4 for further detail. 

A Chi-square test analysis was performed to test if the number of students who passed the course in less than 12 
weeks is the same for each cluster. The distribution of students was statistically different. (Chi-square statistic= 
8.42, p=0.017). It can be concluded that there is evidence to support our claim that the variable mtop predicts 
academic achievement, as the early exit from the course is based on a high score in the coursework as well as in 
the final exam. 

Effect of number ofprogress tests attempted on the final grades. 

The variable Ptest was analyzed to determine if the progress tests administered by the software are supporting 
students’ academic achievement. The descriptive statistics of this variable revealed that the minimum number of 
progress tests taken by students was zero, the maximum number was 13. The average number of progress tests 
taken by students in clusters one, two and three are 3.47, 4.21 and 3.83 respectively. The average number of 
progress tests was not statistically different when compared among the three clusters (F=0.378, p=0.48). It can 
be concluded that the number of progress tests taken by students is not associated with value of the indicator 
mtop. There was no statistical evidence to claim that students in different clusters attempted different number of 
progress tests and whether the number of progress tests had any impact on their learning efforts. 

Correlation and ANOVA test 

Further ANOVA test and correlation analysis were carried out to test whether mtop can be considered as a 
predictor of the final exam (FE) and the coursework (CT). The ANOVA test results showed that the mean value 
of coursework marks and final exam marks are different for all three cluster groups. The difference was found to 
be statistically significant at 0.05 level for CT, F(2, 151)= 4.89, p=0.01 and FE, F(2, 151 )=4.28, p=0.019, 
consequently a statistical difference among the three groups. 

Also, a moderately strong positive and statistically significant correlation was found between the value of mean 
mtop and the marks in the final exam (r=0.41, n=152, p=0.000). From the results of ANOVA test and 
correlation analysis, higher value of mtop indicates higher marks in FE and CT. It can be concluded that the 
ability to study individually is one of the predictors of student’s marks in the coursework and in the final exam. 

Regression analysis 

Since the correlation between mtop and FE is significant, further linear regression analysis was done. The 
unstandardized coefficient for the variable was 69.2, p=0.00 and the constant term is determined as 17.9, p=0.03. 

The value of R 2 is 0.166, which indicates the 16% of the changes in FE are explained by the changes in mtop. 

This implies that there are other predictors which should be explored further. 


Effect of English language proficiency. Out of 152 students, 41 students had a moderate level of English 
language proficiency whereas 111 students had low level of English language proficiency with M=0.63 and 
SD=0.1 compared to those in level 4 who had a mean of M=0.67 and SD=0.12. 

Parametric independent samples t-test was applied to test the hypothesis for research question 3. The output of 
the independent samples t-test (t-value= -2.165 and p-value=0.034) indicates that the mean value of mtop was 
statistically different between the two groups based on their language proficiency level. These results indicate 
that students’ English Language proficiency affects their ability to learn independently. For the current research, 
students’ marks in English were not available for further analysis. 
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Discussion 

On average, students mastered 64% of the topics they practiced. Students in cluster 1 have a higher rate of 
mastering topics whereas students in cluster 3 had a lower rate of mastering topics. Refer to the Figure 2 for 
further detail. 

The high score for mtop can be attributed to the regularity in studying whereas the low score of mastering can be 
due to a lack of time and effort spent at the task. Although some students may be spending sufficient time still 
they may not achieve the expected mastery and such students will have a low score for mtop. Instructors can 
monitor students’ progress periodically and identify students whose score for mtop is less than 0.6. These 
students may need encouragement, motivation as well as additional support to understand a topic. 



Further investigation was done to examine if the distribution of students with different language proficiency is 
uniform across the three clusters. 

Table 5: English language proficiency and cluster membership 


Two-Step Cluster Number 


English level proficiency 

1 (high) 

2(medium) 

3 (Low) 

Total 

Level Low 

Count 

18 

46 

47 

111 


% within Level 

16.2% 

41.4% 

42.3% 

100.0% 

Moderate 

Count 

14 

15 

12 

41 


% within Level 

34.1% 

36.6% 

29.3% 

100.0% 

Total 

Count 

32 

61 

59 

152 


% within Level 

21.1% 

40.1% 

38.8% 

100.0% 


One of the factors affecting students’ ability to learn independently is poor language skills. As shown on Table 
5, only 16% of students with low level English proficiency had a high score for mtop and they belonged to the 
cluster one and 42% had a low score for mtop belonged to cluster 3. 

Another factor affecting the ability to study individually is poor technology skills. Poor technology skills result 
into the under-utilization of the features of ALEKS which is likely to result out of inability to understand gaps in 
what a student knows and what is expected of them. One pertinent theoretical framework is Vygotsky’s concept 
of zone of proximal development. It refers to the gap between ‘what a student can do alone’ and ‘what he can 
achieve with the support from an expert’. The interactions between the expert and the student are termed as 
‘tutorial interactions’ and the expert is termed as the ‘tutor’. The gap between the expert and novice can close 
through scaffolding techniques embedded in the ALEKS software that provide strategies to implement goals of 
constructivist learning paradigm, building on student experience and prior knowledge (Azevedo & Hadwin, 
2005; Hohenwarter, Flohenwarter, Kreis & Lavicza, 2008). 

The most important feature of ALEKS is that it designs a sequence of activities appropriate for each student and 
allows the student to learn at his or her own pace. As a result, it builds confidence in the student to solve 
problems independently. From the findings we can see that the complete potential of ALEKS is utilized, if 
students follow the learning paths suggested by ALEKS. Currently ALEKS interface is not providing clear 
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instructions about how to achieve this and further development may be needed to enhance this aspect of the 
software. 

According to the knowledge space theory, a student is not able to solve problems unless he or she has mastered 
the pre-requisite topics. The limitation in the application is there are no clear instructions presented on the home 
screen of the system. Students often misinterpret this representation as not to complete those topics. This can be 
avoided with an improved representation and menu friendly system, in which the student can see the list of all 
topics without a hyperlink to their detailed explanation. 

Occasional progress tests are administered by ALEKS to detect where students are and generally to formatively 
assess where they can move forward. After each of these formative assessments benchmarks or progress tests, the 
previous learning score is adjusted. This mechanism provides accurate and up to date model of student’s learning 
progress. Students tend to avoid the automatic progress tests and they request teachers to cancel it. It may be due 
to these reasons: the system does not provide details and feedback about the solution submitted during automatic 
progress tests and they have to relearn all topics which are not retained in the progress test. These tests may affect 
the confidence of students because some questions are taken from the list of topics which a student have not yet 
mastered, however, but the system finds that the student is ready to learn. Weaker students fail to answer these 
questions which result in decreasing their previous achievement score. In order to remove these barriers in 
learning independently, students should be given more training about how to use the system. Also quizzes and 
homework assignments on ALEKS can be set as formative assessments as the system provides feedback on these 
assessments unlike the progress tests. 

Conclusion 

In this paper, we established the ratio of topics mastered to topics practiced (mtop) as an indicator of student’s 
ability to study individually. This indicator was further applied for classifying students. This classification 
formed three groups of students for which the mean mtop were 0.80, 0.66 and 0.53 respectively. A strong 
positive and significant correlation was found between the mtop and final exam marks and between mtop and the 
coursework marks, which indicates that mtop can be a predictor of student’s final marks in the ALEKS based 
course. 

Based on the evaluation of the system, we found that ALEKS can measure student’s attainment of factual and 
procedural knowledge, but it fails to measure meta-cognitive aspects, because neither ALEKS shows the 
different strategies to solve these problems nor students could show the strategies used for problem solving in 
order to develop metacognitive abilities. Thus, the ALEKS based coursework can be supplemented by project 
assignments but may require instructional feedback from the instructor. The expectation is that the performance 
effect size of such type of sophisticated intelligent tutors is almost equal to that of expert human tutor (VanLehn 
, 2011 ), 

Limitation and Future Direction 

Though this is a quantitative study and the results are significant, some limitations of this study must be 
considered before generalizing the results. It should be noted that the participants in this study are all female 
students studying English as their second language and this is their first year of using English as the medium of 
instructions. Male students or students with higher proficiency in English may have different ways of learning 
using the tutor. Considering also that learning may be different for a different type of intelligent tutor, results can 
be generalized only on similar population and similar type of intelligent tutor. 

The application of intelligent tutors may not provide the same result to all students. There are other factors, such 
as learning style, efforts, cognitive agility, the affective state of learner and ability to learn using the technology, 
which may have different achievements. Though effectiveness of intelligent tutors has been confinned by many 
researchers, some researchers claim that if students believe a computer can’t help them learn (even though they 
do actually learn), then they have a high probability of disliking the system. They may believe that they cannot 
learn from the tool and may become less motivated to use the tool for learning (Jackson, Graesser & McNamara, 
2009). In continuation of this study, other non-cognitive factors such as students’ learning styles; their attitude 
towards technology and towards mathematics will be analyzed along cognitive and educational elements related 
achievement to understand the impact of these factors on students’ learning experience. 
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